The BRC is: Basic BRC plug.
Every dataset is different. Sometimes very different.
There are many ways to do things. Everyone has their favorite syntax.
The issue:
Many fundamental data processing functions exist in Base R and beyond. Sometimes they can be inconsistent or unnecessarily complex. The result is code that is confusing and doesn’t flow i.e. nested functions
Tidyverse is most importantly a philosophy for data analysis that more often then not makes wrangling data easier. The tidyverse community have built what they describe as an “opinionated” group of packages. These packages readily talk to one another.
Other tools have now been made for the tidy community. This community also overlaps with bioconductor. But the packages above are the linchpins that hold it together.
Workflow Image for working with data.
## salmon_id common_name age_classbylength length_mm IGF1_ng_ml
## 1 35032 Chinook salmon yearling 147 41.26006
## 2 35035 Sockeye salmon juvenile 121 NA
## 3 35036 Sockeye salmon juvenile 112 NA
## 4 35037 Steelhead juvenile 220 42.70981
## 5 35038 Steelhead juvenile 152 NA
## 6 35033 Chinook salmon mixed age juvenile 444 62.11528
## salmon_id common_name age_classbylength variable value
## 1 35032 Chinook salmon yearling length_mm 147.00000
## 2 35032 Chinook salmon yearling IGF1_ng_ml 41.26006
## 3 35033 Chinook salmon mixed age juvenile length_mm 444.00000
## 4 35033 Chinook salmon mixed age juvenile IGF1_ng_ml 62.11528
## 5 35034 Sockeye salmon juvenile length_mm 139.00000
## 6 35034 Sockeye salmon juvenile IGF1_ng_ml NA
## salmon_id common_name age_classbylength variable value
## 1 35032 Chinook salmon yearling length_mm 147
## 2 35033 Chinook salmon mixed age juvenile length_mm 444
## 3 35034 Sockeye salmon juvenile length_mm 139
## 4 35035 Sockeye salmon juvenile length_mm 121
## 5 35036 Sockeye salmon juvenile length_mm 112
## 6 35037 Steelhead juvenile length_mm 220
## salmon_id common_name age_classbylength variable value
## 1 35032 Chinook salmon yearling IGF1_ng_ml 41.26006
## 2 35033 Chinook salmon mixed age juvenile IGF1_ng_ml 62.11528
## 3 35034 Sockeye salmon juvenile IGF1_ng_ml NA
## 4 35035 Sockeye salmon juvenile IGF1_ng_ml NA
## 5 35036 Sockeye salmon juvenile IGF1_ng_ml NA
## 6 35037 Steelhead juvenile IGF1_ng_ml 42.70981
A tidy dataset is a data frame (or table) for which the following are true:
Our first dataframe is tidy
Consistent dataframe layouts help to ensure that all values are present and that relationships between data points are clear.
R is a vectorized programming language. R builds dataframes from vectotrs, and R works best when its operation are vectorized. Tidy data utilizes of both of these aspects of R.
=> Precise and Fast
## ── Attaching packages ───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────── tidyverse 1.2.1 ──
## ✔ ggplot2 3.2.1 ✔ purrr 0.3.3
## ✔ tibble 2.1.3 ✔ dplyr 0.8.3
## ✔ tidyr 1.0.0 ✔ stringr 1.4.0
## ✔ readr 1.3.1 ✔ forcats 0.4.0
## ── Conflicts ──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────── tidyverse_conflicts() ──
## ✖ dplyr::filter() masks stats::filter()
## ✖ dplyr::lag() masks stats::lag()
-> insert graphic here https://github.com/trinker/tidyr_in_a_nutshell
Select allows you to make a vector or dataframe from a specific variable or variables
## # A tibble: 97 x 1
## common_name
## <chr>
## 1 Chinook salmon
## 2 Sockeye salmon
## 3 Sockeye salmon
## 4 Steelhead
## 5 Steelhead
## 6 Chinook salmon
## 7 Sockeye salmon
## 8 Steelhead
## 9 Steelhead
## 10 Steelhead
## # … with 87 more rows
# Select two variables (age_classbylength and common_name)
select(df1, age_classbylength, common_name)## # A tibble: 97 x 2
## age_classbylength common_name
## <chr> <chr>
## 1 yearling Chinook salmon
## 2 juvenile Sockeye salmon
## 3 juvenile Sockeye salmon
## 4 juvenile Steelhead
## 5 juvenile Steelhead
## 6 mixed age juvenile Chinook salmon
## 7 juvenile Sockeye salmon
## 8 juvenile Steelhead
## 9 juvenile Steelhead
## 10 juvenile Steelhead
## # … with 87 more rows
## # A tibble: 97 x 4
## salmon_id common_name age_classbylength IGF1_ng_ml
## <dbl> <chr> <chr> <dbl>
## 1 35032 Chinook salmon yearling 41.3
## 2 35035 Sockeye salmon juvenile NA
## 3 35036 Sockeye salmon juvenile NA
## 4 35037 Steelhead juvenile 42.7
## 5 35038 Steelhead juvenile NA
## 6 35033 Chinook salmon mixed age juvenile 62.1
## 7 35034 Sockeye salmon juvenile NA
## 8 35048 Steelhead juvenile 24.2
## 9 35049 Steelhead juvenile NA
## 10 35050 Steelhead juvenile 63.5
## # … with 87 more rows
# Select all a range of contiguous varibles (common_name:length_mm)
select(df1, common_name:length_mm)## # A tibble: 97 x 3
## common_name age_classbylength length_mm
## <chr> <chr> <dbl>
## 1 Chinook salmon yearling 147
## 2 Sockeye salmon juvenile 121
## 3 Sockeye salmon juvenile 112
## 4 Steelhead juvenile 220
## 5 Steelhead juvenile 152
## 6 Chinook salmon mixed age juvenile 444
## 7 Sockeye salmon juvenile 139
## 8 Steelhead juvenile 288
## 9 Steelhead juvenile 190
## 10 Steelhead juvenile 283
## # … with 87 more rows
Filter allows you to access observations based on specific criteria
# Filter all observation where the variable common_name is Sockeye salmon
filter(df1, common_name == 'Sockeye salmon')## # A tibble: 11 x 5
## salmon_id common_name age_classbylength length_mm IGF1_ng_ml
## <dbl> <chr> <chr> <dbl> <dbl>
## 1 35035 Sockeye salmon juvenile 121 NA
## 2 35036 Sockeye salmon juvenile 112 NA
## 3 35034 Sockeye salmon juvenile 139 NA
## 4 35144 Sockeye salmon juvenile 140 NA
## 5 35147 Sockeye salmon juvenile 115 NA
## 6 35096 Sockeye salmon juvenile 115 NA
## 7 35097 Sockeye salmon juvenile 110 NA
## 8 35098 Sockeye salmon juvenile 112 NA
## 9 35099 Sockeye salmon juvenile 111 NA
## 10 35100 Sockeye salmon juvenile 118 NA
## 11 35119 Sockeye salmon juvenile 122 NA
# Filter all observations where the variable common_name is either Sockeye salmon or Chinook Salmon
filter(df1, common_name %in% c('Sockeye salmon', 'Chinook salmon'))## # A tibble: 57 x 5
## salmon_id common_name age_classbylength length_mm IGF1_ng_ml
## <dbl> <chr> <chr> <dbl> <dbl>
## 1 35032 Chinook salmon yearling 147 41.3
## 2 35035 Sockeye salmon juvenile 121 NA
## 3 35036 Sockeye salmon juvenile 112 NA
## 4 35033 Chinook salmon mixed age juvenile 444 62.1
## 5 35034 Sockeye salmon juvenile 139 NA
## 6 35142 Chinook salmon yearling 149 66.5
## 7 35143 Chinook salmon yearling 204 80.9
## 8 35144 Sockeye salmon juvenile 140 NA
## 9 35145 Chinook salmon yearling 130 23.4
## 10 35146 Chinook salmon mixed age juvenile 422 101.
## # … with 47 more rows
# Filter all observations where the variable common_name ends with 'salmon'. To do this we use stringr function str_ends recognise strings that end with 'salmon'.
filter(df1, str_ends(common_name, 'salmon'))## # A tibble: 59 x 5
## salmon_id common_name age_classbylength length_mm IGF1_ng_ml
## <dbl> <chr> <chr> <dbl> <dbl>
## 1 35032 Chinook salmon yearling 147 41.3
## 2 35035 Sockeye salmon juvenile 121 NA
## 3 35036 Sockeye salmon juvenile 112 NA
## 4 35033 Chinook salmon mixed age juvenile 444 62.1
## 5 35034 Sockeye salmon juvenile 139 NA
## 6 35142 Chinook salmon yearling 149 66.5
## 7 35143 Chinook salmon yearling 204 80.9
## 8 35144 Sockeye salmon juvenile 140 NA
## 9 35145 Chinook salmon yearling 130 23.4
## 10 35146 Chinook salmon mixed age juvenile 422 101.
## # … with 49 more rows
# Filter all observations where the variable length_mm is greater than 200 or less than 120
filter(df1, length_mm > 200 | length_mm < 120)## # A tibble: 36 x 5
## salmon_id common_name age_classbylength length_mm IGF1_ng_ml
## <dbl> <chr> <chr> <dbl> <dbl>
## 1 35036 Sockeye salmon juvenile 112 NA
## 2 35037 Steelhead juvenile 220 42.7
## 3 35033 Chinook salmon mixed age juvenile 444 62.1
## 4 35048 Steelhead juvenile 288 24.2
## 5 35050 Steelhead juvenile 283 63.5
## 6 35051 Steelhead juvenile 279 61.2
## 7 35052 Steelhead juvenile 235 30.6
## 8 35053 Steelhead juvenile 230 49.4
## 9 35056 Steelhead juvenile 208 57.4
## 10 35057 Steelhead juvenile 240 20.2
## # … with 26 more rows
Arrange sorts the dataframe based on a specific variable or variables
## # A tibble: 97 x 5
## salmon_id common_name age_classbylength length_mm IGF1_ng_ml
## <dbl> <chr> <chr> <dbl> <dbl>
## 1 35095 Chinook salmon subyearling 90 NA
## 2 35097 Sockeye salmon juvenile 110 NA
## 3 35099 Sockeye salmon juvenile 111 NA
## 4 35036 Sockeye salmon juvenile 112 NA
## 5 35098 Sockeye salmon juvenile 112 NA
## 6 35147 Sockeye salmon juvenile 115 NA
## 7 35096 Sockeye salmon juvenile 115 NA
## 8 35100 Sockeye salmon juvenile 118 NA
## 9 35035 Sockeye salmon juvenile 121 NA
## 10 35119 Sockeye salmon juvenile 122 NA
## # … with 87 more rows
# Arrange the data first based on the variable common_name, then secondly based on length_mm in a descending order.
arrange(df1, common_name, desc(length_mm))## # A tibble: 97 x 5
## salmon_id common_name age_classbylength length_mm IGF1_ng_ml
## <dbl> <chr> <chr> <dbl> <dbl>
## 1 35033 Chinook salmon mixed age juvenile 444 62.1
## 2 35146 Chinook salmon mixed age juvenile 422 101.
## 3 35110 Chinook salmon mixed age juvenile 275 81.5
## 4 35129 Chinook salmon yearling 225 72.7
## 5 35103 Chinook salmon yearling 216 81.2
## 6 35115 Chinook salmon yearling 215 53.5
## 7 35112 Chinook salmon yearling 205 90.5
## 8 35143 Chinook salmon yearling 204 80.9
## 9 35079 Chinook salmon yearling 199 53.2
## 10 35081 Chinook salmon yearling 196 5.56
## # … with 87 more rows
Mutate creates a new variable based on some form of computation
# A new variable is created based on the caluclation of the z-score of the variable IGF1_ng_ml using scale()
mutate(df1, scale(IGF1_ng_ml, center = TRUE, scale = TRUE))## # A tibble: 97 x 6
## salmon_id common_name age_classbyleng… length_mm IGF1_ng_ml `scale(IGF1_ng_…
## <dbl> <chr> <chr> <dbl> <dbl> <dbl>
## 1 35032 Chinook sal… yearling 147 41.3 -0.258
## 2 35035 Sockeye sal… juvenile 121 NA NA
## 3 35036 Sockeye sal… juvenile 112 NA NA
## 4 35037 Steelhead juvenile 220 42.7 -0.191
## 5 35038 Steelhead juvenile 152 NA NA
## 6 35033 Chinook sal… mixed age juven… 444 62.1 0.704
## 7 35034 Sockeye sal… juvenile 139 NA NA
## 8 35048 Steelhead juvenile 288 24.2 -1.04
## 9 35049 Steelhead juvenile 190 NA NA
## 10 35050 Steelhead juvenile 283 63.5 0.766
## # … with 87 more rows
# A new variable is created called IGFngml_zscore, based on the caluclation of the z-score of the variable IGF1_ng_ml using scale()
mutate(df1, IGFngml_zscore = scale(IGF1_ng_ml, center = TRUE, scale = TRUE))## # A tibble: 97 x 6
## salmon_id common_name age_classbyleng… length_mm IGF1_ng_ml IGFngml_zscore[…
## <dbl> <chr> <chr> <dbl> <dbl> <dbl>
## 1 35032 Chinook sal… yearling 147 41.3 -0.258
## 2 35035 Sockeye sal… juvenile 121 NA NA
## 3 35036 Sockeye sal… juvenile 112 NA NA
## 4 35037 Steelhead juvenile 220 42.7 -0.191
## 5 35038 Steelhead juvenile 152 NA NA
## 6 35033 Chinook sal… mixed age juven… 444 62.1 0.704
## 7 35034 Sockeye sal… juvenile 139 NA NA
## 8 35048 Steelhead juvenile 288 24.2 -1.04
## 9 35049 Steelhead juvenile 190 NA NA
## 10 35050 Steelhead juvenile 283 63.5 0.766
## # … with 87 more rows
Summarize applies aggregating or summary function to a group
# First we define the common_name as a group.
df1_byname <- group_by(df1, common_name)
# Summarise is used to count over the grouped common_names
summarise(df1_byname, count = n())## # A tibble: 4 x 2
## common_name count
## <chr> <int>
## 1 Chinook salmon 46
## 2 Coho salmon 2
## 3 Sockeye salmon 11
## 4 Steelhead 38
# Summarise is used to calculate mean IGF1_ng_ml over the grouped common_names
summarise(df1_byname, IGF1_ng_ml_ave = mean(IGF1_ng_ml, na.rm = T))## # A tibble: 4 x 2
## common_name IGF1_ng_ml_ave
## <chr> <dbl>
## 1 Chinook salmon 46.8
## 2 Coho salmon 73.6
## 3 Sockeye salmon NaN
## 4 Steelhead 46.1
Grouping can also help ask questions with other functions
# Filter obsevrations with the 2 smallest length_mm for each grouped common_names
filter(df1_byname, rank(length_mm) <= 2)## # A tibble: 8 x 5
## # Groups: common_name [4]
## salmon_id common_name age_classbylength length_mm IGF1_ng_ml
## <dbl> <chr> <chr> <dbl> <dbl>
## 1 35038 Steelhead juvenile 152 NA
## 2 35055 Steelhead juvenile 123 55.7
## 3 35145 Chinook salmon yearling 130 23.4
## 4 35085 Coho salmon yearling 140 NA
## 5 35087 Coho salmon yearling 164 73.6
## 6 35095 Chinook salmon subyearling 90 NA
## 7 35097 Sockeye salmon juvenile 110 NA
## 8 35099 Sockeye salmon juvenile 111 NA
## # A tibble: 95 x 5
## # Groups: common_name [3]
## salmon_id common_name age_classbylength length_mm IGF1_ng_ml
## <dbl> <chr> <chr> <dbl> <dbl>
## 1 35032 Chinook salmon yearling 147 41.3
## 2 35035 Sockeye salmon juvenile 121 NA
## 3 35036 Sockeye salmon juvenile 112 NA
## 4 35037 Steelhead juvenile 220 42.7
## 5 35038 Steelhead juvenile 152 NA
## 6 35033 Chinook salmon mixed age juvenile 444 62.1
## 7 35034 Sockeye salmon juvenile 139 NA
## 8 35048 Steelhead juvenile 288 24.2
## 9 35049 Steelhead juvenile 190 NA
## 10 35050 Steelhead juvenile 283 63.5
## # … with 85 more rows
# A new variable is created using z-score within the grouped common_names
mutate(df1_byname, IGFngml_zscore = scale(IGF1_ng_ml, center = TRUE, scale = TRUE))## # A tibble: 97 x 6
## # Groups: common_name [4]
## salmon_id common_name age_classbylength length_mm IGF1_ng_ml IGFngml_zscore
## <dbl> <chr> <chr> <dbl> <dbl> <dbl>
## 1 35032 Chinook salm… yearling 147 41.3 -0.236
## 2 35035 Sockeye salm… juvenile 121 NA NA
## 3 35036 Sockeye salm… juvenile 112 NA NA
## 4 35037 Steelhead juvenile 220 42.7 -0.176
## 5 35038 Steelhead juvenile 152 NA NA
## 6 35033 Chinook salm… mixed age juveni… 444 62.1 0.652
## 7 35034 Sockeye salm… juvenile 139 NA NA
## 8 35048 Steelhead juvenile 288 24.2 -1.12
## 9 35049 Steelhead juvenile 190 NA NA
## 10 35050 Steelhead juvenile 283 63.5 0.890
## # … with 87 more rows
Piping allows you to pass the result from one expression directly into another.
-> same graphic as before , but extend https://github.com/trinker/tidyr_in_a_nutshell
# Without pipe
df1_byname <- group_by(df1, common_name)
summarise(df1_byname, IGF1_ng_ml_ave=mean(IGF1_ng_ml, na.rm=T))## # A tibble: 4 x 2
## common_name IGF1_ng_ml_ave
## <chr> <dbl>
## 1 Chinook salmon 46.8
## 2 Coho salmon 73.6
## 3 Sockeye salmon NaN
## 4 Steelhead 46.1
## # A tibble: 4 x 2
## common_name IGF1_ng_ml_ave
## <chr> <dbl>
## 1 Chinook salmon 46.8
## 2 Coho salmon 73.6
## 3 Sockeye salmon NaN
## 4 Steelhead 46.1
# (1) Group by common_name
# (2) Filter to all those that have length bigger then 200
# (3) Summarise is used to calculate mean IGF1_ng_ml over the grouped common_names for these larger fish
group_by(df1, common_name) %>%
filter(length_mm > 200) %>%
summarise(IGF1_ng_ml_ave = mean(IGF1_ng_ml, na.rm = T))## # A tibble: 2 x 2
## common_name IGF1_ng_ml_ave
## <chr> <dbl>
## 1 Chinook salmon 77.9
## 2 Steelhead 45.3
# (1) Create new variable that is discrete label depending on size of the fish
# (2) Group by common_name and size
# (3) Summarise is used to calculate mean IGF1_ng_ml over the grouped common_names and sizes
mutate(df1, size = if_else(length_mm > 200, 'big_fish', 'small_fish')) %>%
group_by(common_name, size) %>%
summarise(IGF1_ng_ml_ave = mean(IGF1_ng_ml, na.rm = T))## # A tibble: 6 x 3
## # Groups: common_name [4]
## common_name size IGF1_ng_ml_ave
## <chr> <chr> <dbl>
## 1 Chinook salmon big_fish 77.9
## 2 Chinook salmon small_fish 39.5
## 3 Coho salmon small_fish 73.6
## 4 Sockeye salmon small_fish NaN
## 5 Steelhead big_fish 45.3
## 6 Steelhead small_fish 47.3
# (1) Create new variable that is discrete label depending on size of the fish
# (2) Group by common_name and size
# (3) Summarise is used to calculate mean IGF1_ng_ml over the grouped common_names and sizes
# (4) Filter out Coho and Sockeye salmon
mutate(df1, size = if_else(length_mm > 200, 'big_fish', 'small_fish')) %>%
group_by(common_name, size) %>%
summarise(IGF1_ng_ml_ave = mean(IGF1_ng_ml, na.rm = T)) %>%
filter(common_name != 'Coho salmon') %>%
filter(common_name !='Sockeye salmon')## # A tibble: 4 x 3
## # Groups: common_name [2]
## common_name size IGF1_ng_ml_ave
## <chr> <chr> <dbl>
## 1 Chinook salmon big_fish 77.9
## 2 Chinook salmon small_fish 39.5
## 3 Steelhead big_fish 45.3
## 4 Steelhead small_fish 47.3
library(plotly)
p<-mutate(df1,size=if_else(length_mm>200, 'big_fish', 'small_fish')) %>%
group_by(common_name, size) %>%
summarise(IGF1_ng_ml_ave=mean(IGF1_ng_ml, na.rm=T)) %>%
filter(common_name!='Coho salmon') %>%
filter(common_name!='Sockeye salmon') %>%
ggplot( aes(x = common_name, y = IGF1_ng_ml_ave, group = size, fill = size)) +
geom_bar(stat = "identity", position = "dodge") +
theme(axis.text.x = element_text(angle = 90))
ggplotly(p)CHALLENGE
ANSWER
So we blasted through what being tidy can give you. Now lets tidy some data. First step is to read in data.
ReadR:
#Base gives you everything
untidy_counts_base <- read.csv("~/Documents/Box Sync/RU/Teaching/teaching/tidyR/dataset/hemato_rnaseq_counts.csv")
head(untidy_counts_base)## ENTREZ CD34_1 ORTHO_1 CD34_2 ORTHO_2
## 1 350 204 0 103 0
## 2 351 15586 479 10476 39
## 3 353 842 355 1188 86
## 4 354 0 0 0 0
## 5 355 123 291 139 16
## 6 356 1 1 0 0
#ReadR gives you a tibble
untidy_counts <- read_csv("~/Documents/Box Sync/RU/Teaching/teaching/tidyR/dataset/hemato_rnaseq_counts.csv")## Parsed with column specification:
## cols(
## ENTREZ = col_double(),
## CD34_1 = col_double(),
## ORTHO_1 = col_double(),
## CD34_2 = col_double(),
## ORTHO_2 = col_double()
## )
## # A tibble: 100 x 5
## ENTREZ CD34_1 ORTHO_1 CD34_2 ORTHO_2
## <dbl> <dbl> <dbl> <dbl> <dbl>
## 1 350 204 0 103 0
## 2 351 15586 479 10476 39
## 3 353 842 355 1188 86
## 4 354 0 0 0 0
## 5 355 123 291 139 16
## 6 356 1 1 0 0
## 7 357 380 3 177 0
## 8 358 572 2225 597 4051
## 9 359 0 12 1 0
## 10 360 320 502 46 1114
## # … with 90 more rows
#Tibbles carry and display extra information. While reading in it is easy to specify datatype.
untidy_counts <- read_csv("~/Documents/Box Sync/RU/Teaching/teaching/tidyR/dataset/hemato_rnaseq_counts.csv", col_types = cols(
ENTREZ = col_character(),
CD34_1 = col_integer(),
ORTHO_1 = col_integer(),
CD34_2 = col_integer(),
ORTHO_2 = col_integer()
))
untidy_counts## # A tibble: 100 x 5
## ENTREZ CD34_1 ORTHO_1 CD34_2 ORTHO_2
## <chr> <int> <int> <int> <int>
## 1 350 204 0 103 0
## 2 351 15586 479 10476 39
## 3 353 842 355 1188 86
## 4 354 0 0 0 0
## 5 355 123 291 139 16
## 6 356 1 1 0 0
## 7 357 380 3 177 0
## 8 358 572 2225 597 4051
## 9 359 0 12 1 0
## 10 360 320 502 46 1114
## # … with 90 more rows
## # A tibble: 100 x 1
## ENTREZ
## <chr>
## 1 350
## 2 351
## 3 353
## 4 354
## 5 355
## 6 356
## 7 357
## 8 358
## 9 359
## 10 360
## # … with 90 more rows
## # A tibble: 1 x 5
## ENTREZ CD34_1 ORTHO_1 CD34_2 ORTHO_2
## <chr> <int> <int> <int> <int>
## 1 350 204 0 103 0
## [1] "350" "351" "353" "354" "355" "356" "357" "358" "359" "360" "361" "362"
## [13] "363" "364" "366" "367" "368" "369" "372" "373" "374" "375" "377" "378"
## [25] "379" "381" "382" "383" "384" "387" "388" "389" "390" "391" "392" "393"
## [37] "394" "395" "396" "397" "398" "399" "400" "401" "402" "403" "405" "406"
## [49] "407" "408" "409" "410" "411" "412" "414" "415" "416" "417" "419" "420"
## [61] "421" "427" "429" "430" "432" "433" "434" "435" "440" "443" "444" "445"
## [73] "460" "462" "463" "466" "467" "468" "471" "472" "473" "474" "475" "476"
## [85] "477" "478" "479" "480" "481" "482" "483" "486" "487" "488" "489" "490"
## [97] "491" "492" "493" "495"
#Can also not specify which dimension you pull from. This will default to grabbing the column
untidy_counts[1]## # A tibble: 100 x 1
## ENTREZ
## <chr>
## 1 350
## 2 351
## 3 353
## 4 354
## 5 355
## 6 356
## 7 357
## 8 358
## 9 359
## 10 360
## # … with 90 more rows
#All the prior outputs have been outputting another tibble. If double brackets are used a vectr is returned
untidy_counts[[1]]## [1] "350" "351" "353" "354" "355" "356" "357" "358" "359" "360" "361" "362"
## [13] "363" "364" "366" "367" "368" "369" "372" "373" "374" "375" "377" "378"
## [25] "379" "381" "382" "383" "384" "387" "388" "389" "390" "391" "392" "393"
## [37] "394" "395" "396" "397" "398" "399" "400" "401" "402" "403" "405" "406"
## [49] "407" "408" "409" "410" "411" "412" "414" "415" "416" "417" "419" "420"
## [61] "421" "427" "429" "430" "432" "433" "434" "435" "440" "443" "444" "445"
## [73] "460" "462" "463" "466" "467" "468" "471" "472" "473" "474" "475" "476"
## [85] "477" "478" "479" "480" "481" "482" "483" "486" "487" "488" "489" "490"
## [97] "491" "492" "493" "495"
## # A tibble: 100 x 5
## ENTREZ CD34_1 ORTHO_1 CD34_2 ORTHO_2
## <int> <int> <int> <int> <int>
## 1 350 204 0 103 0
## 2 351 15586 479 10476 39
## 3 353 842 355 1188 86
## 4 354 0 0 0 0
## 5 355 123 291 139 16
## 6 356 1 1 0 0
## 7 357 380 3 177 0
## 8 358 572 2225 597 4051
## 9 359 0 12 1 0
## 10 360 320 502 46 1114
## # … with 90 more rows
#Once it is a tibble it is straightforward to modify the datatype
untidy_counts_base <- as_tibble(untidy_counts_base) %>%
mutate_at(vars(ENTREZ), as.character)
untidy_counts_base## # A tibble: 100 x 5
## ENTREZ CD34_1 ORTHO_1 CD34_2 ORTHO_2
## <chr> <int> <int> <int> <int>
## 1 350 204 0 103 0
## 2 351 15586 479 10476 39
## 3 353 842 355 1188 86
## 4 354 0 0 0 0
## 5 355 123 291 139 16
## 6 356 1 1 0 0
## 7 357 380 3 177 0
## 8 358 572 2225 597 4051
## 9 359 0 12 1 0
## 10 360 320 502 46 1114
## # … with 90 more rows
#Some tools are not tibble friendly. Calling as.data.frame is sufficient to convert it back to a base data frame
as.data.frame(untidy_counts_base)## ENTREZ CD34_1 ORTHO_1 CD34_2 ORTHO_2
## 1 350 204 0 103 0
## 2 351 15586 479 10476 39
## 3 353 842 355 1188 86
## 4 354 0 0 0 0
## 5 355 123 291 139 16
## 6 356 1 1 0 0
## 7 357 380 3 177 0
## 8 358 572 2225 597 4051
## 9 359 0 12 1 0
## 10 360 320 502 46 1114
## 11 361 0 1 0 0
## 12 362 3 1 15 0
## 13 363 14 6 4 1
## 14 364 7 0 1 0
## 15 366 6 0 1 0
## 16 367 42 0 51 1
## 17 368 28 0 24 0
## 18 369 1204 1034 833 478
## 19 372 2829 1864 2741 771
## 20 373 179 728 148 795
## 21 374 76 5 138 2
## 22 375 4428 6697 4970 4328
## 23 377 3170 314 2576 11
## 24 378 1839 4845 1767 2975
## 25 379 178 1 181 0
## 26 381 1617 574 1159 339
## 27 382 2874 2265 1746 1668
## 28 383 63 1632 40 721
## 29 384 148 10977 118 94
## 30 387 8899 2457 7405 1228
## 31 388 12598 171 5090 70
## 32 389 2709 193 2313 5
## 33 390 1004 0 395 0
## 34 391 1038 577 1176 164
## 35 392 1527 304 786 71
## 36 393 2949 138 1540 3
## 37 394 1525 464 1062 134
## 38 395 348 67 123 0
## 39 396 6503 702 4723 169
## 40 397 12997 410 11265 38
## 41 398 0 0 0 0
## 42 399 223 11 422 2
## 43 400 1188 147 806 56
## 44 401 0 0 0 0
## 45 402 504 218 496 80
## 46 403 289 25 166 4
## 47 405 1481 824 1004 812
## 48 406 295 175 87 35
## 49 407 4 1 2 1
## 50 408 2451 111 1523 6
## 51 409 2480 1819 1356 226
## 52 410 433 197 215 77
## 53 411 829 217 441 131
## 54 412 312 45 138 17
## 55 414 516 15 396 9
## 56 415 20 0 13 0
## 57 416 2 1 4 0
## 58 417 0 0 0 0
## 59 419 6 5 2 7
## 60 420 141 1136 94 1217
## 61 421 213 255 93 208
## 62 427 4699 11889 1729 926
## 63 429 0 0 2 0
## 64 430 34 13 22 0
## 65 432 63 0 55 1
## 66 433 38 0 26 0
## 67 434 1 1 2 0
## 68 435 408 151 284 34
## 69 440 157 2520 111 535
## 70 443 5 0 4 0
## 71 444 1583 151 747 14
## 72 445 116 15 90 1
## 73 460 34 0 68 0
## 74 462 70 0 31 1
## 75 463 1244 118 492 71
## 76 466 538 480 393 218
## 77 467 2506 402 2130 18
## 78 468 7991 6132 5307 1883
## 79 471 1272 771 1392 53
## 80 472 1389 628 739 138
## 81 473 4173 783 1901 776
## 82 474 0 0 0 0
## 83 475 284 83 467 34
## 84 476 4952 3453 4202 1416
## 85 477 13 3 26 2
## 86 478 78 121 67 0
## 87 479 17 0 4 0
## 88 480 9 5 11 1
## 89 481 1937 18 1017 34
## 90 482 157 1392 75 1660
## 91 483 1075 1454 1789 1141
## 92 486 47 0 18 0
## 93 487 29 33 19 3
## 94 488 4529 1118 2925 269
## 95 489 3465 153 3188 8
## 96 490 1610 1263 913 665
## 97 491 12 1 4 0
## 98 492 4 0 6 0
## 99 493 5011 3585 3053 743
## 100 495 0 0 0 0
# Lets load in some packages
library(org.Hs.eg.db)
library(TxDb.Hsapiens.UCSC.hg19.knownGene)
#Lets use the ENTREZ ID as a key
keys <- untidy_counts$ENTREZ
#We can use the ENTREZ ID to look up Gene Symbol
symbols <- select(org.Hs.eg.db, keys=keys,columns="SYMBOL", keytype="ENTREZID")
#We can use the ENTREZ ID to look up the chormosome the gene resides on
chrs <- select(TxDb.Hsapiens.UCSC.hg19.knownGene, keys=keys, columns="TXCHROM", keytype="GENEID")
#We can use the ENTREZ ID to get a list of genes with grange of their exons
geneExons <- exonsBy(TxDb.Hsapiens.UCSC.hg19.knownGene,by="gene")[keys]
#We will then use an apply to get the transcript length from each gene in the list. The transcript length is calculated by first flattening overalpping exons with reduce(), then calculating the length of each exon with width(), then summing upthe total exon length to get our transcript length.
txsLength <- sapply(geneExons, function(x){ x %>% reduce() %>% width() %>% sum() })
# FInally we have all this metadata. Lets put it together into a tibble.
counts_metadata <- tibble(ID=symbols$ENTREZID,SYMBOL=symbols$SYMBOL,CHR=chrs$TXCHROM,LENGTH=txsLength)## # A tibble: 100 x 5
## ENTREZ CD34_1 ORTHO_1 CD34_2 ORTHO_2
## <chr> <int> <int> <int> <int>
## 1 350 204 0 103 0
## 2 351 15586 479 10476 39
## 3 353 842 355 1188 86
## 4 354 0 0 0 0
## 5 355 123 291 139 16
## 6 356 1 1 0 0
## 7 357 380 3 177 0
## 8 358 572 2225 597 4051
## 9 359 0 12 1 0
## 10 360 320 502 46 1114
## # … with 90 more rows
A single variable with multiple columns
#Pivot longer allows you to collapse variables single varibles that are spread over multiple columns
tidier_counts<-pivot_longer(untidy_counts, cols=c(-ENTREZ), names_to = c("Sample"), values_to = "counts")
#Pivot wider allows you to spread single variables over multiple columns
pivot_wider(tidier_counts, names_from = c("Sample"), values_from = "counts")## # A tibble: 100 x 5
## ENTREZ CD34_1 ORTHO_1 CD34_2 ORTHO_2
## <chr> <int> <int> <int> <int>
## 1 350 204 0 103 0
## 2 351 15586 479 10476 39
## 3 353 842 355 1188 86
## 4 354 0 0 0 0
## 5 355 123 291 139 16
## 6 356 1 1 0 0
## 7 357 380 3 177 0
## 8 358 572 2225 597 4051
## 9 359 0 12 1 0
## 10 360 320 502 46 1114
## # … with 90 more rows
## # A tibble: 400 x 3
## ENTREZ Sample counts
## <chr> <chr> <int>
## 1 350 CD34_1 204
## 2 350 ORTHO_1 0
## 3 350 CD34_2 103
## 4 350 ORTHO_2 0
## 5 351 CD34_1 15586
## 6 351 ORTHO_1 479
## 7 351 CD34_2 10476
## 8 351 ORTHO_2 39
## 9 353 CD34_1 842
## 10 353 ORTHO_1 355
## # … with 390 more rows
Multiple variables in a single column
How do we get tidy? - Cleaning up
#Seperate allows you to break a strings in a vraible by a seperator. In this case the cell type and replicate number are broken by underscore
tidier_counts <- separate(tidier_counts, Sample, sep = "_", into=c("CellType","Rep"), remove=TRUE)
tidier_counts## # A tibble: 400 x 4
## ENTREZ CellType Rep counts
## <chr> <chr> <chr> <int>
## 1 350 CD34 1 204
## 2 350 ORTHO 1 0
## 3 350 CD34 2 103
## 4 350 ORTHO 2 0
## 5 351 CD34 1 15586
## 6 351 ORTHO 1 479
## 7 351 CD34 2 10476
## 8 351 ORTHO 2 39
## 9 353 CD34 1 842
## 10 353 ORTHO 1 355
## # … with 390 more rows
#Unite can go the other way if you want to generate a key. We can use this to make sure we have a Key.
unite(tidier_counts, Sample, CellType, Rep, remove=FALSE)## # A tibble: 400 x 5
## ENTREZ Sample CellType Rep counts
## <chr> <chr> <chr> <chr> <int>
## 1 350 CD34_1 CD34 1 204
## 2 350 ORTHO_1 ORTHO 1 0
## 3 350 CD34_2 CD34 2 103
## 4 350 ORTHO_2 ORTHO 2 0
## 5 351 CD34_1 CD34 1 15586
## 6 351 ORTHO_1 ORTHO 1 479
## 7 351 CD34_2 CD34 2 10476
## 8 351 ORTHO_2 ORTHO 2 39
## 9 353 CD34_1 CD34 1 842
## 10 353 ORTHO_1 ORTHO 1 355
## # … with 390 more rows
Inner Join * Keeps all observations in x and y with matching keys
Outer Join * A left join keeps all observations in x and those in y with matching keys. * A right join keeps all observations in y and those in x with matching keys. * A full join keeps all observations in x and y
# In this pipe I group by gene, summarise the data based on the sum of counts, and filter for anything that has a count greater than 0.
expressed_genes <- tidy_counts %>%
group_by(ENTREZ) %>%
summarise(count_total=sum(counts)) %>%
filter(count_total>0)
# Left join shows all genes as my full data frame tidy_counts is used as the backbone. Thefiltered expressed genes is secondary, and has missing values (unexpressed genes) whichare filled with NA
left_join(tidy_counts, expressed_genes, by = c("ENTREZ" = "ENTREZ")) %>% print(n=20)## # A tibble: 400 x 6
## ENTREZ Sample CellType Rep counts count_total
## <chr> <chr> <chr> <chr> <int> <int>
## 1 350 CD34_1 CD34 1 204 307
## 2 350 ORTHO_1 ORTHO 1 0 307
## 3 350 CD34_2 CD34 2 103 307
## 4 350 ORTHO_2 ORTHO 2 0 307
## 5 351 CD34_1 CD34 1 15586 26580
## 6 351 ORTHO_1 ORTHO 1 479 26580
## 7 351 CD34_2 CD34 2 10476 26580
## 8 351 ORTHO_2 ORTHO 2 39 26580
## 9 353 CD34_1 CD34 1 842 2471
## 10 353 ORTHO_1 ORTHO 1 355 2471
## 11 353 CD34_2 CD34 2 1188 2471
## 12 353 ORTHO_2 ORTHO 2 86 2471
## 13 354 CD34_1 CD34 1 0 NA
## 14 354 ORTHO_1 ORTHO 1 0 NA
## 15 354 CD34_2 CD34 2 0 NA
## 16 354 ORTHO_2 ORTHO 2 0 NA
## 17 355 CD34_1 CD34 1 123 569
## 18 355 ORTHO_1 ORTHO 1 291 569
## 19 355 CD34_2 CD34 2 139 569
## 20 355 ORTHO_2 ORTHO 2 16 569
## # … with 380 more rows
# Right join shows only genes that survived filtering as it is using the second dataframe as the backbone for the new dataframe.
tidy_counts_expressed <- right_join(tidy_counts,expressed_genes, by = c("ENTREZ" = "ENTREZ"))
tidy_counts_expressed %>% print(n=20)## # A tibble: 376 x 6
## ENTREZ Sample CellType Rep counts count_total
## <chr> <chr> <chr> <chr> <int> <int>
## 1 350 CD34_1 CD34 1 204 307
## 2 350 ORTHO_1 ORTHO 1 0 307
## 3 350 CD34_2 CD34 2 103 307
## 4 350 ORTHO_2 ORTHO 2 0 307
## 5 351 CD34_1 CD34 1 15586 26580
## 6 351 ORTHO_1 ORTHO 1 479 26580
## 7 351 CD34_2 CD34 2 10476 26580
## 8 351 ORTHO_2 ORTHO 2 39 26580
## 9 353 CD34_1 CD34 1 842 2471
## 10 353 ORTHO_1 ORTHO 1 355 2471
## 11 353 CD34_2 CD34 2 1188 2471
## 12 353 ORTHO_2 ORTHO 2 86 2471
## 13 355 CD34_1 CD34 1 123 569
## 14 355 ORTHO_1 ORTHO 1 291 569
## 15 355 CD34_2 CD34 2 139 569
## 16 355 ORTHO_2 ORTHO 2 16 569
## 17 356 CD34_1 CD34 1 1 2
## 18 356 ORTHO_1 ORTHO 1 1 2
## 19 356 CD34_2 CD34 2 0 2
## 20 356 ORTHO_2 ORTHO 2 0 2
## # … with 356 more rows
Show diagram of the math
tidy_counts_expressed <- tidy_counts_expressed %>% group_by(Sample) %>% mutate(CPM=(counts/sum(counts))*1000000)
tidy_counts_expressed <- tidy_counts_expressed %>% inner_join(counts_metadata, by = c("ENTREZ" = "ID")) %>% mutate(TPM=(counts/(LENGTH/1000))/(sum(counts)/1000000))
p <- tidy_counts_expressed %>% ggplot(aes(x=CPM, y=TPM)) +geom_point() + scale_x_continuous(name="log2(CPM)",trans='log2') + scale_y_continuous(name="log2(TPM)",trans='log2')
ggplotly(p)## Warning: Transformation introduced infinite values in continuous x-axis
## Warning: Transformation introduced infinite values in continuous y-axis
readr – reading data into R dplyr – mainpulating data tibble - working with tibbles tidyr – miscellaneous tools for tidying data ggplot2 – making pretty graphs stringr – working with strings purr - iterating over data forcats - working with factors
#stringr
Character manipulation: these functions allow you to manipulate individual characters within the strings in character vectors.
Whitespace tools to add, remove, and manipulate whitespace.
Locale sensitive operations whose operations will vary from locale to locale.
Pattern matching functions. These recognise four engines of pattern description. The most common is regular expressions, but there are three other tools.
#purr
applying functions to datasets
Hadley Wickham (Chief Scientist at RStudio) has been the driving force behind the tidyverse. It now serves as the basis for many R packages and functions.
There is a lot of support at https://www.tidyverse.org/
Hadley also wrote a paper about why tidy data is best: www.jstatsoft.org/v59/i10/paper.